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Abstract 

Background: This study aims to rank policy concerns and policy-related research issues in order to identify policy 
and research gaps on access to medicines (ATM) in low- and middle-income countries in Latin America and the 
Caribbean (LAC), as perceived by policy makers, researchers, NGO and international organization representatives, 
as part of a global prioritization exercise. 

Methods: Data collection, conducted between January and May 201 1, involved face-to-face interviews in El 
Salvador, Colombia, Dominican Republic, and Suriname, and an e-mail survey with key-stakeholders. Respondents 
were asked to choose the five most relevant criteria for research prioritization and to score policy/research items 
according to the degree to which they represented current policies, desired policies, current research topics, and/or 
desired research topics. Mean scores and summary rankings were obtained. Linear regressions were performed 
to contrast rankings concerning current and desired policies (policy gaps), and current and desired research 
(research gaps). 

Results: Relevance, feasibility, and research utilization were the top ranked criteria for prioritizing research. Technical 
capacity, research and development for new drugs, and responsiveness, were the main policy gaps. Quality 
assurance, staff technical capacity, price regulation, out-of-pocket payments, and cost containment policies, were 
the main research gaps. There was high level of coherence between current and desired policies: coefficients of 
determination (R 2 ) varied from 0.46 {Health system structure; r = 0.68, P <0.01) to 0.86 {Sustainable financing; r = 0.93, 
P <0.01). There was also high coherence between current and desired research on Rational selection and use of 
medicines {r = 0.71, P <0.05, R 2 = 0.51), Pricing/ affordability {r = 0.82, P <0.01, R 2 = 0.67), and Sustainable financing 
{r = 0.76, P <0.01, R 2 = 0.58). Coherence was less for Health system structure {r = 0.61, P <0.01, R 2 = 0.38). 

Conclusions: This study combines metrics approaches, contributing to priority setting methodology development, 
with country and regional level stakeholder participation. Stakeholders received feedback with the results, and we 
hope to have contributed to the discussion and implementation of ATM research and policy priorities in LAC. 

Keywords: Access to medicines, Health systems, Health systems research, Interviews, Latin America and the 
Caribbean, Priority setting, Web survey 
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Background 

The achievement of the health-related Millennium De- 
velopment Goals [1] strongly depends on health system 
capacity. Research for health has been recognized as 
crucial to generate innovations driven by health prob- 
lems and diseases that disproportionally affect low- 
and middle-income countries (LMICs), as well as to 
provide the scientific and technical evidence needed 
to promote decision-making effectiveness regarding re- 
source allocation, policies, and health system strengthe- 
ning interventions [2-6]. 

Since 1990, the Commission on Health Research for De- 
velopment identified a huge inequity in health research fi- 
nancing, with only 4.4% of global resources directed to 
health problems of developing countries. By 2000, this per- 
sistent imbalance between spending and disease burden 
was characterized as the 10/90 gap' [7,8]. International 
political declarations call for increased and more equitable 
funding for research, and to make it more responsive to 
health systems needs in LMICs [3,4,7,9]. In this sense, set- 
ting research priorities at national, regional, and global 
levels is a core strategy aimed at closing the gap [5,7,10]. 

The delivery of medicines to people in need is an essen- 
tial component to strengthen health systems effectiveness. 
Ensuring access to medicines (ATM) depends on efforts re- 
lated to different domains such as sustainable financing, 
the existence of a network of reliable health services, and 
an efficient supply chain management [11,12]. Therefore, it 
is crucial to generate reliable and accurate information on 
different ATM domains that could be used for strengthen- 
ing health systems. Yet, in order to make research respon- 
sive to health systems needs in LMICs, priority-setting 
processes can be useful for targeting research that presents 
the greatest potential public health benefit [10]. 

As part of a global exercise on priority setting for 
health policy and research on ATM in LMICs, coor- 
dinated by the Alliance for Health Policy and Systems 
Research (AHPSR) [13], two studies involving Latin 
American and the Caribbean (LAC) LMICs were carried 
out by researchers from the Sergio Arouca National 
School of Public Health, Brazil. One of them, published 
in 2013 [14], consisted of a scoping study that identified 
ATM research issues and gaps through the analysis of 
the scientific publication trends in LAC. It showed that, 
even though a lot of effort has been made and achieve- 
ments have been reached in some countries, research on 
ATM still remains a challenge for countries in Latin 
America and the Caribbean. The second study, presen- 
ted in this paper, aimed at ranking policy/research items, 
in order to identify policy and research gaps on ATM in 
LMICs of LAC, as perceived by policy makers, resear- 
chers, and non-governmental organization (NGO) and 
international organization representatives; the results of 
both were used in the global exercise [13]. 



The main objectives of the AHPSR global study were 
to identify priority policy issues in ATM relevant for 
LMICs, to identify research questions that would help 
address these policy issues, and to prioritize these re- 
search questions in a health policy and systems research 
(HPSR) agenda [13]. Different researchers have studied 
LMICs from other geographic regions. 

Methods 

The data collection strategy conducted from January to 
May 2011 involved two approaches: face-to-face inter- 
views with key-stakeholders from selected countries and 
an e-mail survey. Both approaches used the same ques- 
tionnaire, in which a single list of items was ranked. Re- 
spondents were asked to score each policy/research item 
according to the degree to which it represented a cur- 
rent policy, a desired policy, a current research topic, 
and/or a desired research topic. 

The methodology adopted in this study was adapted 
from Urcullo et al. [15], whose method was applied in a 
similar study on priority setting for HPSR, on health fi- 
nancing and human resources for health. 

Research settings and participants selection 

Face-to-face interviews were carried out in four countries: 
El Salvador; Colombia, Dominican Republic, and Suriname. 
Country selection criteria covered the inclusion of both 
lower- (El Salvador) and upper-middle income countries 
(Colombia, Dominican Republic, and Suriname) from all 
Pan American Health Organization (PAHO) sub-regions 
(Central America, South America, and the Caribbean); 
existence of local researchers working on ATM and related 
issues; and agreement from country representatives to par- 
ticipate in the study. As medicines were seen as a sensitive 
political subject, PAHO officers mediated the identification 
of countries and the invitation to join the study. 

In each country, interviews with 12 stakeholders were 
planned: five government decision makers (two high 
level decision makers at the Ministry of Health (MoH); 
one person in charge of pharmaceuticals at the MoH; 
one member of the national congress health commis- 
sion; one person from the drug regulatory authority); 
one member of medical and one member of pharmaceu- 
tical professional associations; one representative of in- 
ternational organizations (World Health Organization/ 
PAHO; USAID; Inter- American Development Bank; World 
Bank); two researchers working on ATM; and two NGO 
representatives. Local partners from PAHO offices sup- 
ported the identification of interviewers. 

The e-mail survey was carried out to expand the cover- 
age of countries and key stakeholders. It was planned to 
include one person in charge of pharmaceuticals in the 
MoH and two activists from NGOs with relevant actions 
on ATM from each LMIC in the region (excluding the 
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visited ones); experts with at least three publications on 
ATM in LAC in the previous 10 years; 23 international 
agency representatives (PAHO focal points on medicines); 
and persons from the pharmaceutical industry with ATM 
projects in the region. We were able to identify 112 poten- 
tial respondents. After sending the first invitation, two 
follow-up contacts were made requesting a reply. 

Questionnaire structure 

The first step in the development of the questionnaire 
was the selection of a list of items that would be sub- 
jected to ranking. Items were generated as exhaustively 
as possible and were phrased as to reflect both policy is- 
sues and research topics, pertinent to the discussion of 
ATM. Those policy/research items were developed in an 
AHPSR expert meeting, held in Cambodia in 2010 - a 
preparatory meeting for regional prioritization exercises, 
which were intended to subsidize a global prioritization 
effort on ATM research [13]. Regional team leaders ag- 
reed upon an analytical framework based on the WHO 
four ATM dimensions (rational selection and use, af- 
fordable prices, sustainable financing, and reliable health 
and supply systems) and determinants rooted in local, 
national, and international contexts [12]. For this regio- 
nal exercise, items generated were organized in a matrix 
reflecting the analytical framework (Table 1). 

The questionnaire was composed of five parts: I) inter- 
viewee profile (background, educational level and affili- 
ation); II) an open-ended question regarding access to 
medicines; III) ranking of research prioritization criteria; 
IV) policy priority rankings; and, V) research priority 
rankings. This paper discusses parts III, IV and V. 

In part III, interviewees and respondents were pro- 
vided with a list of prioritization criteria for research and 
asked to rank them, in an attempt to identify what 
should drive research priority definitions on ATM. It 
also elucidates which criteria were more likely to guide 
respondents' rankings of the research alternatives. 

In part IV, interviewees and respondents were pro- 
vided with a list of items (Table 1) and asked to rank 
them, as they would represent policy priorities regarding 
the following aspects: 

• Policies being carried out by the government 
(current policies); 

• Policies that the government should be carrying out 
(desired policies). 

In part V, interviewees and respondents were asked to 
rank the same list of items, but in this case, as they would 
reflect research priorities on ATM: 

• Topics being investigated at national research 
centers (current research); 



• Research topics required for public health policies 
(desired research). 

In all ranking parts of the questionnaire, items could 
be added (by means of an "other" option) if participants 
felt that the provided list was incomplete. 

All research materials were produced in Portuguese 
and translated into English and Spanish. For face-to-face 
interviews, a printed list of items was given to the inter- 
viewees in order to help them think about each item. 
Trained members of the research team, who performed 
pre-tests and meetings to standardize questionnaire appli- 
cation, conducted the interviews. The e-mail survey ver- 
sion contained explanations and instructions to assure a 
common comprehension and filling of the questionnaire. 

Ranking and scoring procedures 

Interviewees and e-mail survey respondents were asked 
to choose the five most relevant criteria for research prio- 
ritization (part III) and policy/research items (parts IV 
and V), and sort them from 1 (top priority) to 5 (5 th prio- 
rity). Selected criteria and policy/research items (Table 1) 
received a score for each interviewee/respondent (from 5 
to 1, with higher scores for the most important and lower 
scores for the least important within the chosen list). Cri- 
teria and policy/research items that were not among the 
five most relevant received a score of 0. 

We calculated average scores for each country visited 
as well as for the e-mail survey responses. Total scores 
for each criterion and summary rankings for each as- 
sessed aspect (current and desired policies; current and 
desired research) were obtained by summing those ave- 
rage scores. 

Coherence analysis 

A coherence analysis was performed in order to answer 
the following questions: 

1. How well does the government respond to the needs 
of the population? - By comparing answers about 
policies being carried out by the government 
(current policies) vs. policies that the government 
should be carrying out (desired policies). 

2. Is scientific evidence being generated to support 
policies that respond to the needs of the population? - 
By comparing answers about policies that the 
government should be carrying out (desired policies) 
vs. topics being investigated at national research 
centers (current research). 

3. Is research moving in the right direction? - By 
comparing answers about topics being investigated 
at national research centers (current research) vs. 
research topics required for public health policies 
(desired research). 



Table 1 Policy/research items* generated by the regional meeting in Cambodia according to the World Health Organization framework for access to medicines 
and Health System Level** 

WHO Framework for ATM 



1. Rational selection and use of medicines 



2. Pricing/affordability 



3. Reliable health system/health 
system structure 



4. Sustainable financing 



I. Individual, household, 
and community level 



II. Public and private 
health service delivery 
channels 



, Health sector 



1a) Information asymmetry (user knowledge level; 
communication and language barriers - as for 
ethnic minorities) 

1 b) Health seeking behavior, preference for 

private care, preference for secondary level of 
care (bypass primary health care), self- 
medication 

1c) Beliefs about illness and treatment (traditional 
practices; demand for injections and branded 
medicines; prescribers and dispensers 
perception of quality of drugs) 

Id) Impact of advertisements on medicines; 
incentives (or lack of, leading to private or 
dual practice); medicines becoming a source 
in financing for health services 

1e) Staff capacity for rational prescription and use 
of medicines; training curriculum 

If) Clinical treatment guidelines, and essential 
medicines list: development; incentives for 
implementation; operational mechanisms; 
standardization (including between private 
and public sectors) 

1g) Incentives for rational use of medicines and 
for implementation of generic policy 



1 h) Staff deployment 



1i) Pharmacovigilance, information on adverse 
drug reaction, and other problems related to 
medicines 



1j) Medicine information system (not only price 
information)/competing with medicines 
advertisement 



2a) High out-of-pocket payment 

2c) Community participation in medicine 
delivery arrangements 



2d) Medicine price variation according to 
geographical location (e.g., urban/rural 
differences, higher prices in remote 
areas); price differential between public 
and private 



2b) Opportunity costs 

2e) Medicine price information system 

2f) Impact of prices on access 

2g) Policy and regulation for medicines 
price 

2h) Incentives for implementation of 
generic policy 



3a) Geographical accessibility: physical 
barriers; distance to facilities; 
remoteness (combination of 
physical barriers and infrastructure 
weakness) 

3e) Traditional medicine (e.g., unclear 
distinction between allopathy and 
non-allopathy) 



3b) Distribution systems and supply 
chains 

3c) Pharmaceutical services at local 
level 

3d) Informal markets - substandard 
quality and counterfeit medicines 

3f) Staff and technical capacity: for 
supply management; with 
managerial, interpersonal, and 
information technology (IT) skills; 
with local language skills 

3g) Responsiveness to patient needs; 
differential responsiveness 
between public and private 

3j) Availability of medicines (especially 
in public sector) 

3h) Public-private mix: reliance on pri- 
vate sector delivery; public-private 
partnerships and their role in ac- 
cess to medicines 

3i) Central procurement policies vs. 
decentralization 

3k) Quality assurance 

31) Coordination between health 
policies and medicines policies; 
referral policy/referral system 

3m) Monitoring and evaluation 

systems; funding for monitoring 
and evaluation 



4a) Community financing 
arrangements 

4b) Role of government 
subsidies at community 
level 

4c) Focal models of subside 



4d) Health provider payment 
methods 

4e) Health insurance coverage 
and models 



4f) Government budgets for 
medicines 

4g) Reimbursement policies 

4h) Cost containment policies 



Table 1 Policy/research items* generated by the regional meeting in Cambodia according to the World Health Organization framework for access to medicines 
and Health System Level** (Continued) 



IV. Beyond the health 
sector 



1k) Intersectoral initiatives (e.g., rational use of 
medicines in schools - role of Ministry of 
Education) 



2i) Patents and intellectual property issues 

2j) Finance policies: taxes, autonomy, 
privatization; exemption systems 



2k) Trade and economic goals (impact of 
health sector policies outside the health 
sector) 



21) Budget allocation to health 
2m) Medicine production 



3n) Governance and governing: law 
and regulation enforcement- 
transparency and accountability 



4i) Donor funding; 
harmonization and 
alignment; verticalized 
donor support 

3o) Regional integration and economic 4j) Patents and intellectual 



cooperation (example: UNASUL, 
MERCOSUL) 



3p) Promotion of research and 
development - new drugs/ 
neglected diseases. 



3q) Medicine production 



property issues 

4k) Finance policies: taxes, 
autonomy, privatization 

41) Trade and economic goals 
(impact of health sector 
policies outside the health 
sector) 

4m) Budget allocation to 
health 



*ltem codes reflect the order in which they appeared in the questionnaire. Here they are presented according to Health System Level. 

**Health System Levels according to Bigdeli et al. [12]: I. Individual, household and community; II. Health service delivery; III. Health Sector; IV. Beyond the health sector (Public policies cutting across sectors; and, 
International and regional level). 
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Linear regressions depicted on scatter plots were per- 
formed to test these relationships. Strength and direction 
of association were estimated by calculating the cor- 
relation coefficient (Pearsons r) and its statistical sig- 
nificance (P value). Data linearity fit was expressed by 
coefficients of determination (R 2 ), used as coherence 
indicators. 

We assumed that interviewees and respondents are le- 
gitimate spokespersons to indicate priorities on behalf of 
the population, and are informed enough to point out 
policies governments are currently carrying out and 
topics currently being investigated at national research 
centers. 

The National School of Public Health Ethics Research 
Committee approved the project. All interviews were 
conducted after the signing of a written informed con- 
sent. In the case of the e-mail survey, the informed con- 
sent was presented to participants as the first page of 
the form and they were asked to fill it in if they agreed 
to participate. 

Results 

Forty-nine interviews were completed in country visits 
(Table 2). Two of the selected stakeholders were unavail- 
able in Suriname and El Salvador. In Colombia, a higher 
number of stakeholders were included, due to an in- 
creased interest and participation of NGO representa- 
tives during data collection. 

From the target of 112 stakeholders, 98 e-mails were 
successfully sent (14 emails bounced), and a final re- 
sponse rate of approximately 35.7% was reached after at 
least two attempts. The final sample of respondents in 
the e-mail survey was mainly composed of researchers 
(48.6%) and NGO representatives (22.9%) - which ex- 
panded the representativeness of those types of stake- 
holders in the study, followed by international agency 
representatives (mainly PAHO officers; 14.3%), and MoH 
representatives (11.4%). There was a low representative- 
ness of pharmaceutical industry respondents, with only 
one response. 

Research prioritization criteria 

The highest ranked criterion that should be applied 
in the definition of research priorities on ATM was 
Relevance, which was also the most cited in 
Colombia, Dominican Republic, and in the e-mail sur- 
vey (Table 3). The second criterion was Feasibility, 
and the third Research utilization. The latter, was also 
one of the top three ranked criteria in Colombia, 
Suriname, and in the e-mail survey. Whereas the first 
item implies a relationship between the measurement 
or magnitude of a problem and its degree of priority, 
the others refer to more operational or pragmatic as- 
pects of research - denoting expectations towards the 



Table 2 Number of respondents by country 



Information source 



Number of respondents 



Country visits 

Colombia 

Dominican Republic 
El Salvador 
Suriname 
E-mail survey 

Argentina 

Bolivia 

Brazil 

Colombia 

Costa Rica 

Ecuador 

USA 

Guatemala 
Mexico 
Peru 
Total 



49 

15 

12 

11 

11 

35 

3 

1 

13 

3 

1 

6 
4 
1 
1 

2 

84 



Source: Authors. 



usefulness of policy research as an input to public 
management and decision-making. 

In El Salvador, political and rights-based criteria were 
considered more important than managerial ones. These 
criteria were also high ranked in the Dominican Repub- 
lic and in Suriname. Responsiveness to the National 
Health Policy was considered important in the e-mail 
survey (Table 3). 

It is important to notice that criteria were identified 
and ranked for scoring policy relevance of research 
items were used implicitly by participants, and were 
scored separately (i.e., participants gave one overall 
score to each item according to the degree to which 
it was desired research, and not according to each 
criterion). 

Policy and research rankings 

Table 4 shows the three top ranked policy/research 
items in each assessed aspect, as well as their summary 
rankings. 

One single item from the domain of Rational Selection 
and Use of Medicines scored the highest on almost all 
categories (current and desired policies, and current and 
desired research): (If) Clinical treatment guidelines, and 
Essential Medicines List. 

On the other domains, the highest ranked items re- 
garding current policies were also the highest ranked for 
desired policies: (2g) Policy and regulation for medicines 
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Table 3 Respondent rankings of criteria that should be applied to define research priorities on access to medicines 





Items (proposed criteria) 


Colombia 


Dominican 
Republic 


El Salvador 


Suriname 


E-mail 
survey 


Total 


g 


Relevance (magnitude of the problem; persistence of the problem; 
impact on health; urgency) 


1.5 


1.8 


0.6 


0.9 


2.0 


6.7 


b 


Feasibility (capacity of the system to carry out the research; funding 
support; justification of the cost/investment; justification of time) 


0.8 


1.0 


0.8 


0.7 


0.7 


4.1 


i 


Research utilization (adequacy and usefulness of the current 
knowledge base (avoiding duplication); applicability of the research 
outcome; availability of cost-effective interventions (access); 

uptrl clUUl Icll ell cLU vtrl \<cbb) 


1 .0 


0.3 


0.5 


1 .1 


1 .0 


3.9 


f 


Political will/acceptability/commitment 


0.5 


1.1 


0.9 


0.8 


0.5 


3.8 


c 


Human rights issues; equity focus; ethical and moral issues 


0.7 


1.3 


0.9 


0.3 


0.6 


3.7 


h 


Responsiveness to the National Health Policy or national goals 


0.7 


0.8 


0.4 


0.8 


1.0 


3.6 


a 


Community concern/demand; environmental health and 
sociopolitical effects 


0.3 


0.3 


1.1 


0.9 


0.4 


3.0 


d 


Impact on development; economic impact 


0.2 


0.6 


0.5 


0.4 


0.3 


1.9 


e 


Partnership building; research capacity building 


0.3 


0.2 


0.0 


0.1 


0.4 


1.0 



*ltem codes reflect the order in which they appeared in the questionnaire. Here they are presented according to total ranking scores. Bold numbers represent the 
three highest scores in each setting. 
Source: Authors. 



price - Pricing/ Affordability; (3a) Geographical accessi- 
bility - Reliable Health System! Health System Structure; 
and (4f) Government budgets for medicines - Sustain- 
able Financing, 

With respect to current research, the main items were 
(If) clinical treatment guidelines and essential medi- 
cines list; (2f) impact of prices on access; (3j) availability 
of medicines; and (4e) health insurance coverage and 
models. 

In the domain of Rational Selection and Use of Medi- 
cines, the main desired research items were (If) clinical 
treatment guidelines and essential medicines list, (le) 
staff capacity and training, and (li) pharmacovigilance, 
demonstrating an overall demand for investigations that 
focus on health service delivery and health sector levels. 
Regarding Pricing! Affordability, (2f ) impact of prices on 
access, (2g) policy and regulation for medicines price, 
and (2a) high out-of-pocket payment received the high- 
est summary rankings. In the domain of Reliable Health 
System! Health System Structure, the main topics re- 
quired for the design of public health policies were first 
(3k) quality assurance and, second, (3j) availability of me- 
dicines; the third highest scored topic was (3a) geograph- 
ical accessibility. When it comes to Sustainable Financing, 
stakeholders wanted to increase knowledge mainly on 
(4h) cost containment policies, (4e) health insurance co- 
verage and models, and (4m) budget allocation to health. 
This demonstrates the interest in research focused on 
meso- and macro-level issues involved in Sustainable Fi- 
nancing, rather than micro-, health service, or community 
level topics in this domain. 



Coherence analysis 

How well do governments respond to the ATM needs of 
populations? 

Analysis of summary rankings indicates a high level of 
coherence between current and desired policies in all 
domains: coefficients of determination (R 2 ) vary from 
0.46 for Health System Structure (with a high correlation 
coefficient; r = 0.68, P <0.01) to 0.86 for Sustainable Fi- 
nancing (r = 0.93, P <0.01) - current policies, overall, 
would cover adequate topics (Figure 1). 

Despite the general coherence level observed, some 
items (dots on scatter plots) are distant from trend lines 
and should receive special consideration. Dots above trend 
lines represent policies desired by stakeholders but not 
sufficiently addressed by current policies - representing 
policy gaps such as item (le) staff capacity and training 
for Rational Selection and Use of Medicines. The domain 
with the lowest coherence indicator, Health System Struc- 
ture, presented several gaps: (3p) promotion of research 
and development - new drugs/neglected diseases; (3g) re- 
sponsiveness to patients' needs; (3n) governance: law en- 
forcement; transparency and accountability; and (3f) staff 
and technical capacity. 

Is scientific evidence being generated to support policies 
that respond to the needs of the population? 

Research on Rational selection and use of medicines and 
on Sustainable Financing seems to be addressing impor- 
tant policy issues, which is indicated by high coherence 
between current research and desired policies (r = 0.73, 
P <0.05, R 2 = 0.53; r = 0.80, P <0.01, R 2 = 0.65, respectively). 
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Table 4 Summary ranking scores for current policies, desired policies, current research, and desired research 


Item 

codes* 


WHO Framework domain 
Items (policy or research topics) 


Current 
policies 


Desired 
policies 


Current 
research 


Desired 
research 




Rational Selection and Use of Medicines 










It 


Clinical treatment guidelines, and essential medicines list: development; incentives for 
implementation; operational mechanisms; standardization (including between private 
and public sectors) 


1 7.7 


12.4 


9.8 


9.8 


1i 


Pharmacovigilance, information on adverse drug reaction, and other problems related 
to medicines 


10.1 


8.0 


6.9 


7.6 


I e 


Staff capacity for rational prescription and use of medicines; training curriculum 


7.1 


1 1 .5 


D.U 




Id 


Impact of advertisements on medicines; incentives (or lack of, leading to private or 
dual practice); medicines becoming a source in financing for health services 

Pricing/Affordability 


3.3 


7.3 


5.1 


7.0 


2g 


Policy and regulation for medicines price 


11.4 


10.9 


5.7 


9.4 


If 

Z\ 


Impact of prices on access 


O.Z 


o.U 
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However, item (le), Staff capacity and training for rational 
prescription and use of medicines {Rational Selection and 
Use of Medicines), is highly prioritized as a desired policy 
but does not figure amongst the main investigated topics. 
Concerning Sustainable Financing, some items such as 
(4f ) government budgets for medicines and (4h) cost con- 
tainment policies represent highly desired policies, which 
are not investigated as much (Figure 2). 

The lowest coherence is observed for Health System 
Structure (r = 0.49, P <0.05, R 2 = 0.24). Equally important 
policy issues as (3f) staff and technical capacity, (3k) qual- 
ity assurance, and (3j) availability of medicines have diffe- 
rent levels of research intensity, with a predominance of 
the latter (3j). Also, highly prioritized policy issues like 
(3a) geographical accessibility and (3b) distribution sys- 
tems and supply chains received much less attention on 
research. Items (3f), (3a), and (3b) could be considered 



the most important research topics required to support 
policies that respond to the needs of the population in this 
domain. 

Coherence between current research and desired pol- 
icies is moderate for Pricing and Affordability (r = 0.64, 
P <0.05, R 2 = 0.41). In this dimension, (2g) policy and 
regulation for medicines price and (2a) high out-of- 
pocket payment are high priority issues that should be 
addressed by public policies but they are not being in- 
vestigated quite as much. On this domain, research on 
(2i) patent and intellectual property issues is more inten- 
sive, but it does not represent a highly ranked desired 
policy. 

Is research moving in the right direction? 

High coherence between current research and desired re- 
search indicates that topics currently being investigated at 
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Figure 1 Scatter plots between summary rankings for current and desired policies, by WHO Framework domain. Each dot represents a 
policy/research item. 

V J 



national research centers seem to cover important re- 
search topics, required for public health policies on 
the domains of Rational selection and use of medi- 
cines (r = 0.71, P <0.05, R 2 = 0.51), Pricing and Afford- 
ability (r = 0.82, P <0.01, R 2 = 0.67), and Sustainable 
Financing (r = 0.76, P <0.01, R 2 = 0.58) (Figure 3). Staff 
capacity and training for rational prescription and use 
of medicines (le), (2g) policy and regulation for medi- 
cines price, (2a) high out-of-pocket payment, and (4h) 
cost containment policies are highly desired research 
topics that have not being investigated as much, represent- 
ing important research gaps on these domains. 

Health System Structure, the domain with the lowest 
coherence indicator between current and desired research 
(r = 0.61, P <0.01, R 2 = 0.38), presented several gaps: (3k) 



quality assurance, (3a) geographical accessibility, and (3f) 
staff and technical capacity. The latter (3f) presents the 
same level of research desirability as other items (31, 3d, 
and 3b), but received less attention from research initia- 
tives. Distance from trend line makes (3a) the most evi- 
dent research gap on this domain. 

Discussion 

Ranson and Bennett [16] argue that HPSR priorities 
should emerge from priority setting processes, but ad- 
equate processes are rarely performed. They also refer to 
a WHO survey performed in LMICs showing that often 
policy- makers, researchers, and users of research are un- 
aware of how priorities are identified or set. Further- 
more, priority-setting exercises usually do not reflect a 
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Figure 2 Scatter plots between summary rankings for current research and desired policies, by WHO Framework domain. Each dot 
representing a policy/research item. 



health system perspective and are not "bottom-up", i.e., 
does not consider views of stakeholders. This study de- 
parted from the organization of issues pertaining to 
ATM from a health systems perspective [12], and was 
primarily based upon local level stakeholders' inputs. 
Even though policy/research items subject to rankings 
were developed prior to stakeholders consultation, all 
ranking lists included an open-ended "other" option with 
the intention of encouraging respondents to add any issue 
or topic deemed important and not adequately reflected 
on the list. That option was rarely used by respondents, 
which suggests that there was a good level of acceptance 
and endorsement of the list supplied. Additionally, before 



being asked to rank policy/research items, participants 
were asked to expose their concerns on ATM policies and 
research in an open-ended question, most of the topics or 
issues that emerged from those answers were similar to 
items listed in the ranking part of the questionnaire. 

According to Viergever et al. [10], there are two main 
groups of methods that can be applied to set priorities. 
Consensus-based approaches involve putting people to- 
gether and in line, allowing participants to reconsider 
their positions, which tends to improve the acceptability 
of outputs. However, a consensus-based approach would 
be difficult for a regional exercise, and has been per- 
formed for smaller scale exercises only. Metrics-based 
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Figure 3 Scatter plots between summary rankings for current and desired research, by WHO Framework domain. Each dot representing 
a policy/research item. 



approaches, as the one adopted in this study, generate 
results by pooling individual rankings, which averts the 
predominance of few most influential participants. 

We derived data from individual responses, which were 
aggregated statistically, aiming at means and averages and 
looking for communalities rather than disagreements. 
Therefore, there was no opportunity for participants to 
confront their views and possibly change their positions to 
generate consensus. The type of analysis performed is in- 
novative in the field of priority-setting exercises. It es- 
tablishes major research gaps and policy concerns both 
directly through the analysis of summary scores, and 
indirectly through linear regressions. The latter made 
it possible to identify those policy/research items with 



a gap between their degree of perceived implementation 
(current policy/research rankings) and degree of desirabil- 
ity (desired policy/ research rankings). This could only be 
done through the use of a metrics-based approach. 

Tromp and Baltussen [17] suggest that an explicit con- 
ceptual mapping of criteria facilitates decision making. 
Even though they discuss priority setting of health inter- 
ventions, and not health research as focused here, the 
rationale of aggregating criteria in subsets was useful to 
reduce the original list of criteria used in our study, and 
helped define the final list of criteria submitted to re- 
spondents. Although we have not used an explicit con- 
ceptual mapping of criteria, there was an effort to form 
groups of criteria with some shared meaning. Instead of 
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predefining which criteria respondents should use when 
ranking research items, we submitted the prioritization 
exercise to a previous identification of relevant criteria 
according to them. This was an attempt to capture re- 
spondents' normative standpoints towards research on 
ATM. We tried to learn which criteria our pool of inter- 
viewees and respondents consider legitimate and rele- 
vant and were likely to have influenced their responses. 
Also, it was a moment to reflect upon those criteria be- 
fore ranking priorities - which could facilitate decision 
when answering the questionnaire. 

Stakeholder inclusiveness was an important concern of 
this study, as suggested by Viergever [10]. Even though 
we cannot assume that participants were statistically re- 
presentative of all groups involved, we relied on good 
choice of well-informed respondents and a balance among 
the type of stakeholders to form an adequate pool of re- 
spondents. PAHO local officers support was of utmost 
importance in the identification of relevant national stake- 
holders for the face-to-face interviews. 

Viergever et al. [10] state that priority-setting exercises 
should be based on agreed criteria. In our study, despite 
some variation among countries, relevance, feasibility, 
and research utilization were the most cited. They differ 
from those agreed in an international consensus exercise 
on ATM - innovation, impact on health and health sys- 
tems, equity, and lack of research [13]. This may rely on 
participants background and context of each exercise. 
Stakeholders that contributed to our study were asked to 
base their responses on their countries specificities, while 
in the international exercise participants were asked to 
consider the global scenario. 

Setting priorities for health policy interventions is not 
the same as setting priorities for research for health. 
Groups of stakeholders affected by and interested in po- 
licy decisions may differ from those usually interested or 
involved in research on the same subject. Criteria that 
are relevant to define policy intervention priorities also 
differ from those that are adequate to judge how research 
should be prioritized. This study addressed ATM-related 
issues, expressed on the set of items submitted to ranking, 
in which policy interventions can be designed and under 
which research can be developed. As mentioned above, 
this regional study contributed to a global initiative to 
identify research topics under relevant policy issues [13]. 
Thus, important insight on how to proceed was gathered 
mainly from literature on research for health priority 
setting, but also from priority setting on health policy 
interventions. 

Clinical treatment guidelines and essential medicines 
lists, pertaining to Rational Selection and Use of Medi- 
cines, received the highest scores, indicating that this is 
perceived as both an important current policy issue and 
a highly desired one. It is perceived as a relevant ATM 



research topic currently being investigated in the region, 
and also a highly desired research topic. Since 1978, 
WHO proposes rational selection as one of four key fac- 
tors influencing ATM, and promotes the adoption of 
essential medicines lists as a backbone for national phar- 
maceutical policies [11,18,19]. This aspect was relevant in 
a broader international research priority exercise [13] and 
figures among the mostly addressed issues in peer re- 
viewed publications in the region [14]. 

Staff capacity and training appeared as an important 
gap, both for policy and for research, and among differ- 
ent ATM domains. The policy gap on staff capacity for 
rational selection and use of medicines is probably re- 
lated to the growing need to adopt evidence-based prac- 
tices [20], which demands skilled staff, generally scarce 
in LMICs. Due to its importance, the health workforce 
is stated as one of the health system building blocks 
[21,22] remaining a worldwide priority and still an import- 
ant gap [9,23]. A scoping study addressing peer reviewed 
publication related to ATM in LAC also highlighted 
human resources and health information as under- 
represented themes in ATM research [14]. 

The need for policies addressing the development of 
new medicines for neglected disease emerged as a gap, 
showing the lack of treatment options for endemic dis- 
eases is still prevalent in the region [24,25]. Tromp and 
Baltussen [17] recognize responsiveness to the popula- 
tion needs as one of the five categories that reflect the 
goals of a health system. In our study it emerged as an 
important policy gap. Another important policy gap, 
good governance, has been increasingly recognized as 
necessary for improving access to medicines and con- 
tributing to health systems strengthening, since corrup- 
tion and low transparency is an important cause of 
wastage of scarce resources [26]. 

It is remarkable that the same issues appearing as 
main gaps between current and desired policies - staff 
capacity and training and cost containment policies - 
also emerged as important research gaps. Geographical 
accessibility and distribution systems and supply chains 
appeared amongst highly desired policy issues that lack 
research. Geographical accessibility figures as one of the 
main research gaps. Many countries in Latin America 
have their population concentrated in big cities, but have 
large extensions of their territories with low population 
density, generally low-income regions, not attractive to 
private medicine suppliers. 

This study presents some limitations. The response 
rate was approximately 36% for the e-mail survey. Ne- 
vertheless, this is consistent with the mean response rate 
(34.6%, SD = 15.7%) reported on a meta-analysis devel- 
oped by Cook et al. [27] for studies with no missing data. 
The over-representativeness of academia and NGOs in 
the e-mail survey was counterbalanced by interviews at 
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country level, where other stakeholder representativeness 
was greater. A purposive sampling of countries was used 
and results may not be generalized to other countries. 

Different from other prioritization exercises on ATM 
[13], we have decided - with regional stakeholders - not 
to detail priorities as specific research questions. Our aim 
was to map priority topics and to sensitize stakeholders 
about the importance of evidence-based decision-making 
and HPSR, motivating them to develop local research pri- 
ority exercises. Thus, our results should not be taken as 
top-down advice, but a support to decision makers on the 
establishment of their own priorities, in which we agree 
withTromp and Baltussen [17]. 

Conclusions 

This study combines strong metrics approaches, such as 
the coherence analysis, which contributes to priority set- 
ting methodology development, with country and regional 
level stakeholders' participation. As part of a more com- 
prehensive approach, which included a scoping study of 
the scientific literature, it achieved useful results both at 
country and regional levels, and also contributed to a glo- 
bal exercise of research priority settings. 

Staff and technical capacity, promotion of Research 
and Development for new drugs for neglected diseases, 
and responsiveness to patients' needs, were the main 
policy gaps. Human resources technical capacity, medi- 
cines price regulation, out-of-pocket payments, cost con- 
tainment policies, and quality assurance, were the main 
research gaps identified. 

In order to give feedback to the participants and in- 
crease the chance of the results being used, an executive 
summary of the final report was shared with the partici- 
pants and PAHO officers. Also, results were presented 
in an internet-based online lecture, with the same audi- 
ence, apart from having been presented in other tech- 
nical and scientific meetings. So far, we have not received 
any feedback on the use of those priorities to support 
decision-making on visited settings. We believe that the 
influence of those exercises should merit itself some space 
in future studies. We hope our study contributes to the 
discussion and implementation of ATM research and pol- 
icy priorities in LAC. 
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